Methods and Systems for Enhanced Automated Visual Inspection of a Physical Asset

ABSTRACT

A computer-implemented system for enhanced automated visual inspection of a physical asset includes a visual inspection device capable of generating images of the physical asset and a computing device including a processor and a memory device coupled to the processor. The computing device includes a storage device coupled to the memory device and coupled to the processor. The storage device includes at least one historic image of the physical asset and at least one engineering model substantially representing the physical asset. The computing device is configured to receive, from a present image source, at least one present image of the physical asset captured by the visual inspection device. The computing device is configured to identify at least one matching historic image corresponding to the at least one present image. The computing device is configured to identify at least one matching engineering model corresponding to the at least one present image.

BACKGROUND

The field of the invention relates generally to computer-implementedprograms and, more particularly, to a computer-implemented system forenhanced automated visual inspection of a physical asset.

Known methods of visual inspection of physical assets include the use ofoptical devices inserted within such physical assets without humanentrance. Such known methods provide benefits by allowing for rapidvisual analysis of complex physical assets that may be inaccessible tohuman technicians without disassembly or other servicing. In some suchknown cases, disassembly or servicing of complex physical assets maytake hours to perform. During such disassembly or servicing, many suchcomplex physical assets must be rendered temporarily inoperable.Therefore, known methods of visual inspection rapidly expedite analysisof internal conditions of complex physical assets and reduce downtimethat may be caused by disassembly or servicing.

Known methods and systems of visual inspection of physical assetsinvolve sending visual data to human technicians capable of diagnosingconditions of the complex physical assets. Human technicians may reviewsuch visual data using monitors, computers, or other displays. In manyknown methods and systems of visual inspection, human technicians willoverlay collected visual data onto a three-dimensional engineering modeland manually match features between the collected visual data and thethree-dimensional model. Such overlaying, comparison, and matching maybe very time consuming. Known methods and systems of visual inspectionmay also face the complexity that when visual data is received by thehuman technician, it is often too late to obtain new physicalmeasurements that have not been previously collected by field engineers.In such cases, new physical measurements may only be collected on thenext inspection of the physical asset.

Such known methods and systems of visual inspection of physical assetsusing optical devices do not facilitate efficient and effectiveidentification of the component corresponding to visual data sent fromthe optical device to the human technician. In many known complexphysical assets there are numerous components and sub-components. Reviewof the architectural details of each component and sub-component isoften a necessary and time-consuming aspect of using known methods ofvisual inspection. Such review is necessary, however, in order to ensurethat the human technician understands the exact component orsub-component associated with visual data. Misidentification may belogistically and financially expensive.

BRIEF DESCRIPTION

In one aspect, a computer-implemented system for enhanced automatedvisual inspection of a physical asset is provided. The system includes avisual inspection device capable of generating images of the physicalasset. The system also includes a computing device. The computing deviceincludes a processor. The computing device also includes a memory devicecoupled to the processor. The computing device also includes a storagedevice coupled to the memory device and additionally coupled to theprocessor. The storage device includes at least one historic image ofthe physical asset and at least one engineering model substantiallyrepresenting the physical asset. The computing device is configured toreceive, from a present image source, at least one present image of thephysical asset captured by the visual inspection device. Also, thecomputing device is configured to identify at least one matchinghistoric image corresponding to the at least one present image. Further,the computing device is configured to identify at least one matchingengineering model corresponding to the at least one present image.

In a further aspect, a computer-based method for enhanced automatedvisual inspection of a physical asset is provided. The computer-basedmethod is performed by a computing device. The computing device includesa processor. The computing device also includes a memory device coupledto the processor. The computing device further includes a storage devicecoupled to the memory device and additionally coupled to the processor.The storage device includes at least one historic image of the physicalasset and at least one engineering model substantially representing thephysical asset. The computer-based method includes receiving, from apresent image source, at least one present image of the physical assetcaptured by the visual inspection device. The computer-based method alsoincludes identifying at least one matching historic image correspondingto the at least one present image. The computer-based method furtherincludes identifying at least one matching engineering modelcorresponding to the at least one present image.

In another aspect, a computer for enhanced automated visual inspectionof a physical asset is provided. The computer includes a processor. Thecomputer also includes a memory device coupled to the processor. Thecomputer further includes a storage device coupled to the memory deviceand also coupled to the processor. The storage device includes at leastone historic image of the physical asset and at least one engineeringmodel substantially representing the physical asset. The computer isconfigured to receive, from a present image source, at least one presentimage of the physical asset captured by the visual inspection device.The computer is also configured to identify at least one matchinghistoric image corresponding to the at least one present image. Thecomputer is further configured to identify at least one matchingengineering model corresponding to the at least one present image.

DRAWINGS

These and other features, aspects, and advantages will become betterunderstood when the following detailed description is read withreference to the accompanying drawings in which like charactersrepresent like parts throughout the drawings, wherein:

FIG. 1 is a schematic view of an exemplary high-levelcomputer-implemented system for enhanced automated visual inspection ofa physical asset;

FIG. 2 is a block diagram of an exemplary computing device that may beused with the computer-implemented system shown in FIG. 1;

FIG. 3 is flow chart of an exemplary process for enhanced automatedvisual inspection of a physical asset using the computer-implementedsystem shown in FIG. 1; and

FIG. 4 is a simplified flow chart of the overall method for enhancedautomated visual inspection of a physical asset using thecomputer-implemented system shown in FIG. 1 to facilitate the enhancedvisual inspection process shown in FIG. 3.

Unless otherwise indicated, the drawings provided herein are meant toillustrate key inventive features of the invention. These key inventivefeatures are believed to be applicable in a wide variety of systemscomprising one or more embodiments of the invention. As such, thedrawings are not meant to include all conventional features known bythose of ordinary skill in the art to be required for the practice ofthe invention.

DETAILED DESCRIPTION

In the following specification and the claims, reference will be made toa number of terms, which shall be defined to have the following meanings

The singular forms “a”, “an”, and “the” include plural references unlessthe context clearly dictates otherwise.

“Optional” or “optionally” means that the subsequently described eventor circumstance may or may not occur, and that the description includesinstances where the event occurs and instances where it does not.

As used herein, the term “non-transitory computer-readable media” isintended to be representative of any tangible computer-based deviceimplemented in any method or technology for short-term and long-termstorage of information, such as, computer-readable instructions, datastructures, program modules and sub-modules, or other data in anydevice. Therefore, the methods described herein may be encoded asexecutable instructions embodied in a tangible, non-transitory, computerreadable medium, including, without limitation, a storage device and/ora memory device. Such instructions, when executed by a processor, causethe processor to perform at least a portion of the methods describedherein. Moreover, as used herein, the term “non-transitorycomputer-readable media” includes all tangible, computer-readable media,including, without limitation, non-transitory computer storage devices,including, without limitation, volatile and nonvolatile media, andremovable and non-removable media such as a firmware, physical andvirtual storage, CD-ROMs, DVDs, and any other digital source such as anetwork or the Internet, as well as yet to be developed digital means,with the sole exception being a transitory, propagating signal.

As used herein, the terms “software” and “firmware” are interchangeable,and include any computer program stored in memory for execution bydevices that include, without limitation, mobile devices, clusters,personal computers, workstations, clients, and servers.

As used herein, the term “real-time” refers to at least one of the timeof occurrence of the associated events, the time of measurement andcollection of predetermined data, the time to process the data, and thetime of a system response to the events and the environment. In theembodiments described herein, these activities and events occursubstantially instantaneously.

As used herein, the term “computer” and related terms, e.g., “computingdevice”, are not limited to integrated circuits referred to in the artas a computer, but broadly refers to a microcontroller, a microcomputer,a programmable logic controller (PLC), an application specificintegrated circuit, and other programmable circuits, and these terms areused interchangeably herein.

As used herein, the term “automated” and related terms, e.g.,“automatic,” refers to the ability to accomplish a task without anyadditional input. Also, as used herein, the visual inspection of aphysical asset is automated from the point that a reference image (e.g.,the present image) is provided to the system to the point that it thereference image is overlaid over a three-dimensional model.

As used herein, the term “visual inspection device” and related terms,e.g., “visual inspection devices,” refers to any optical device capableof being inserted into a physical asset, moving within a physical asset,capturing visual data regarding the physical asset, and transmitting thevisual data to other systems. Such visual inspection devices mayinclude, without limitation, borescopes, fiberscopes, video borescopes,rigid borescopes, or any digital camera capable of being inserted andmaneuvered within a physical asset. Also, as used herein, visualinspection devices facilitate enhanced inspection of a physical asset byproviding present images from the physical asset.

As used herein, the term “physical asset” and related terms, e.g.,“assets,” refers to any physical object that may be inspected using avisual inspection device. Such assets may include, without limitation,gas turbines, steam turbines, aircraft engines, diesel engines,automotive engines, truck engines, pressure vessels, and any simple orcomplex machinery that may be penetrated by a visual inspection device.Also, as used herein, physical assets are used to receive enhancedvisual inspection.

As used herein, the term “engineering model” and related terms, e.g.,“engineering model data,” refers to any three-dimensional graphicalmodel that substantially represents a physical asset. Such engineeringmodels may include, without limitation, computer-aided drawings (CAD),computer-aided industrial design, photo realistic renderings, and anyother model that substantially represents a physical asset and can beused to depict a typical physical asset in three dimensions. Also, asused herein, engineering models are used to describe the expectednormal, three-dimensional design of the physical asset.

As used herein, the term “key point feature detection” and related termsrefers to an approach to detect and describe local features in images.Key point feature detection typically includes methods of identifyinginteresting points on an object, extracting these points as features,and describing the features. Such methods include, without limitation,Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features(SURF), Maximally Stable Extremal Regions (MSER), and Affine-SIFT(ASIFT). As algorithms evolve, some methods may change, be added, or beremoved. Also, as used herein, key point feature detection allows forthe comparison of present images with historic image data.

As used herein, the term “key point matching” and related terms refersto a process conducted after key point feature detection where a firstfeature from a first image can be found to refer to the same physicallocation on the physical asset as a second feature on a second image.Key point matching, therefore, allows for the correlation of featuresdetermined in key point feature identification between two differentimages. Also, as used herein, key point matching facilitates comparisonof present images with historic image data.

As used herein, the term “point cloud” and related terms, e.g., “2D-3Dpoint cloud” refers to a mapping of feature matches between athree-dimensional model and a two-dimensional historic image. Also, asused herein, the point cloud is created in an offline process and usedto provide a three-dimensional engineering model along with thetwo-dimensional historic image after image retrieval identifies thehistoric image corresponding to the present image.

As used herein, the term “image retriever” and related terms, e.g.,“image retrieval system,” refers to a method of finding a second imagethat is most relevant to (i.e., most closely corresponds to) a firstimage. Many methods of image retrieval exist and accordingly the imageretriever may apply any such method. The image retriever may include,without limitation, methods using K-Nearest Neighbor (K-NN), methodsimplementing a vocabulary tree and cluster-based retrieval, or any othermethod capable of organizing historic images and efficiently identifyinghistoric images that correspond to a present image. Also, as usedherein, the image retriever is used to identify a historic image andpoint cloud corresponding to a present image.

As used herein, the term “epipolar constraint” and related terms, e.g.,“epipolar line,” refer to a concept in computer vision and geometry thatallows for the reconciling of two two-dimensional images taken of thesame three-dimensional object from different positions. An epipolar linerefers to the line that is perceived as a point in a first image andline in the second image. The divergent perception is caused by thedifferent reference positions of the first and second image. As usedherein, epipolar lines are used to constrain matching points between twoimages. The epipolar constraint is the use of the epipolar line torequire that for each point observed in one image the same point must beobserved in the other image on a known epipolar line. Where an epipolarconstraint is not satisfied (i.e., the first image presents a pointwhere the proposed corresponding point in the second image does not lieon the corresponding epipolar line) a match is noted as an error andpenalized or discredited with respect to the other matches.

Approximating language, as used herein throughout the specification andclaims, may be applied to modify any quantitative representation thatcould permissibly vary without resulting in a change in the basicfunction to which it is related. Accordingly, a value modified by a termor terms, such as “about” and “substantially”, are not to be limited tothe precise value specified. In at least some instances, theapproximating language may correspond to the precision of an instrumentfor measuring the value. Here and throughout the specification andclaims, range limitations may be combined and/or interchanged, suchranges are identified and include all the sub-ranges contained thereinunless context or language indicates otherwise.

FIG. 1 is a schematic view of an exemplary high-levelcomputer-implemented system 100 for enhanced visual inspection of aphysical asset 105. In the exemplary embodiment, physical asset 105 is asteam turbine. In alternative embodiments, physical asset 105 may be anyphysical object including, without limitation, gas turbines, aircraftengines, diesel engines, automotive engines, truck engines, pressurevessels, piping systems, generators, reduction gears, transformers, andany simple or complex machinery. Also, computer-implemented system 100includes a visual inspection device 110. In the exemplary embodiment,visual inspection device 110 is a borescope. In alternative embodiments,visual inspection device 110 may include, without limitation,fiberscopes, video borescopes, rigid borescopes, or any digital cameradevice capable of being inserted and maneuvered within physical asset105.

Computer-implemented system 100 also includes a computing device 120.Computing device 120 includes a processor 125 and a memory device 130.Processor 125 and memory device 130 are coupled to one another. In theexemplary embodiment, computing device 120 includes one processor 125and one memory device 130. In alternative embodiments, computing device120 may include a plurality of processors 125 and/or a plurality ofmemory devices 130.

Computing device 120 also includes a storage device 135. Storage device135 is coupled to processor 125 and also coupled to memory device 130.In the exemplary embodiment, storage device 135 is a hard disk drive. Inalternative embodiments, storage device 135 may be any storage deviceincluding, without limitation, optical storage devices, magnetic storagedevices, network storage devices, flash storage devices, mass storagedevices, and any other storage device capable of storing data forcomputer-implemented system 100. In at least some embodiments, computingdevice 120 is associated with external storage 160. In such embodiments,external storage 160 stores data that may be written to storage device135 or used directly by computing device 120.

In the exemplary embodiment, storage device 135 includes historic imagedata 140 and engineering model data 145. Historic image data 140represents data captured from prior use of visual inspection device 110to inspect physical asset 105. In the exemplary embodiment, at leastsome historic image data 140 is created immediately after manufacturingphysical asset 105. In alternative embodiments, historic image data 140may be created at any time prior to the current inspection. In otherembodiments, historic image data 140 may include image data frommultiple inspections of physical asset 105. Engineering model data 145represents data of pre-manufacturing designs of physical asset 105. Inthe exemplary embodiment, engineering model data 145 includescomputer-aided drawings (CAD). In alternative embodiments, engineeringmodel data 145 may include, without limitation, computer-aidedindustrial design, photo realistic renderings, and any other model thatsubstantially represents a physical asset and can be used to depicttypical physical asset 105.

In operation, visual inspection device 110 is inserted into physicalasset 105 and transmits at least one present image 115 to computingdevice 120. Computing device 120 receives at least one present image 115from visual inspection device 110. In alternative embodiments, computingdevice 120 may receive at least one present image 115 generated byvisual inspection device 110 but stored at memory device 130, storagedevice 135, or external storage 160. In these embodiments, present imagedata 115 may be received by alternative sources for reasons ofexpediency. For example, the technician using visual inspection device110 may have conducted a thorough inspection of physical asset 105 andcaptured all relevant image data 115 but is no longer available tostream live present image data 115. In such examples, accessing the samepresent image data 115 from data storage may be valuable.

Also, computing device 120 identifies at least one matching historicimage (not shown in FIG. 1) from historic image data 140 correspondingto at least one present image 115. Further, computing device 120identifies at least one matching engineering model (not shown in FIG. 1)from engineering model data 145 corresponding to at least one presentimage 115. In the exemplary embodiment, historic image data 140corresponding to present image 115 is retrieved by an image retrievalprocess along with engineering model data 145 through the use of a pointcloud which matches features between two-dimensional historic image data140 and three-dimensional engineering model data 145. Point clouds andthe relationship between two-dimensional historic image data 140 andthree-dimensional engineering model data 145 are generated in a separateprocess from what is described in this disclosure. In the exemplaryembodiment, the image retrieval process involves the use of a spatialentropy feature which captures the inter-region and intra-regioncharacteristics based upon entropy. The spatial entropy featurerepresents the specific pose of visual inspection device 110 and scalesimilarities. The spatial entropy features are used to build an entropyfeature vector which uses information regarding a given pixel and itsneighbors (i.e., the pixels immediately adjacent to the pixel). Thespatial nature of the image is represented by dividing the image intospatial cells and assigning a value for entropy to each cell andconcatenating the entropy values of all cells into a feature vector.

In some embodiments, computing device 120 may measure and comparefeatures from present image 115 and engineering model data 145 todetermine variances between physical asset 105 and engineering modeldata 145. Variances may include, for example, a crack on the surface ofphysical asset 105, a warping of a component of physical asset 105, orany other condition that may cause a variance between present image 115and engineering model data 145. Computing device 120 transmits anyvariances to a servicer 155 capable of providing diagnostic,replacement, and/or maintenance services to physical asset 105. In suchembodiments, servicer 155 is capable of receiving information fromcomputing device 120 regarding physical asset 105.

FIG. 2 is a block diagram of exemplary computing device 120 that may beused with computer-implemented system 100 (shown in FIG. 1). Computingdevice 120 includes a memory device 130 and a processor 125 operativelycoupled to memory device 130 for executing instructions. In theexemplary embodiment, computing device 120 includes a single processor125 and a single memory device 130. In alternative embodiments,computing device 120 may include a plurality of processors 125 and/or aplurality of memory devices 130. In some embodiments, executableinstructions are stored in memory device 130. Computing device 120 isconfigurable to perform one or more operations described herein byprogramming processor 125. For example, processor 125 may be programmedby encoding an operation as one or more executable instructions andproviding the executable instructions in memory device 130.

In the exemplary embodiment, memory device 130 is one or more devicesthat enable storage and retrieval of information such as executableinstructions and/or other data. Memory device 130 may include one ormore tangible, non-transitory computer-readable media, such as, withoutlimitation, random access memory (RAM), dynamic random access memory(DRAM), static random access memory (SRAM), a solid state disk, a harddisk, read-only memory (ROM), erasable programmable ROM (EPROM),electrically erasable programmable ROM (EEPROM), and/or non-volatile RAM(NVRAM) memory. The above memory types are exemplary only, and are thusnot limiting as to the types of memory usable for storage of a computerprogram.

Memory device 130 may be configured to store operational data including,without limitation, at least one present image 115 (shown in FIG. 1),and/or any other type of data. In some embodiments, processor 125removes or “purges” data from memory device 130 based on the age of thedata. For example, processor 125 may overwrite previously recorded andstored data associated with a subsequent time and/or event. In addition,or alternatively, processor 125 may remove data that exceeds apredetermined time interval. Also, memory device 130 includes, withoutlimitation, sufficient data, algorithms, and commands to facilitateoperation of computer-implemented system 100.

In some embodiments, computing device 120 includes a user inputinterface 230. In the exemplary embodiment, user input interface 230 iscoupled to processor 125 and receives input from user 225. User inputinterface 230 may include, without limitation, a keyboard, a pointingdevice, a mouse, a stylus, a touch sensitive panel, including, e.g.,without limitation, a touch pad or a touch screen, and/or an audio inputinterface, including, e.g., without limitation, a microphone. A singlecomponent, such as a touch screen, may function as both a display deviceof presentation interface 220 and user input interface 230.

A communication interface 235 is coupled to processor 125 and isconfigured to be coupled in communication with one or more otherdevices, such as a sensor or another computing device 120, and toperform input and output operations with respect to such devices. Forexample, communication interface 235 may include, without limitation, awired network adapter, a wireless network adapter, a mobiletelecommunications adapter, a serial communication adapter, and/or aparallel communication adapter. Communication interface 235 may receivedata from and/or transmit data to one or more remote devices. Forexample, a communication interface 235 of one computing device 120 maytransmit an alarm to communication interface 235 of another computingdevice 120. Communications interface 235 facilitates machine-to-machinecommunications, i.e., acts as a machine-to-machine interface.

Presentation interface 220 and/or communication interface 235 are bothcapable of providing information suitable for use with the methodsdescribed herein, e.g., to user 225 or another device. Accordingly,presentation interface 220 and communication interface 235 may bereferred to as output devices. Similarly, user input interface 230 andcommunication interface 235 are capable of receiving informationsuitable for use with the methods described herein and may be referredto as input devices. In the exemplary embodiment, presentation interface220 is used to visualize and display the automatic overlay of presentimages 115 on engineering model data 145. Once visualized user 225 mayuse user input interface 230 to execute tasks including, withoutlimitation, measurements and determinations of variances in presentimages 115 with respect to engineering model data 145. The task ofmeasurement may include the use of software which provides measuringapplications to make such measurements and determinations of variances.

In the exemplary embodiment, user 225 may use computing device 120 byreceiving information on at least one present image 115, at least onematching historic image (not shown), and at least one matchingengineering model (not shown) via presentation interface 220. User 225may act on the information presented and use computing device 120 toupdate the condition of physical asset 105, request services (not shown)from servicer 155, or continue inspection of physical asset 105. User225 may initiate such an action via user input interface 230 whichprocesses the user command at processor 125 and uses communicationinterface 235 to communicate with other devices.

In the exemplary embodiment, computing device 120 is an exemplaryembodiment of computing device 120. In at least some other embodiments,computing device 120 is also an exemplary embodiment of other devices(not shown) used for enhanced visual inspection of physical asset 105.In most embodiments, computing device 120 at least illustrates theprimary design of such other devices.

FIG. 3 is flow chart of an exemplary process 300 for enhancing visualinspection of physical asset 105 using computer-implemented system 100(both shown in FIG. 1). Process 300 includes receiving 305 presentvisual inspection device data. Receiving 305 present visual inspectiondevice data represents the receipt at computing device 120 (shown inFIG. 1) of present images 115 (shown in FIG. 1) from visual inspectiondevice 110 (shown in FIG. 1) capturing images from physical asset 105.

Process 300 additionally uses the image retriever to retrieve a pointcloud 315. In the exemplary embodiment, the image retriever uses aK-Nearest Neighbor search to search through a kd-tree structurecontaining historic image data 140. In alternative embodiments, theimage retriever uses different methods to search for a correspondinghistoric image 140. The point cloud represents a mapping ofthree-dimensional engineering model data 145 to two-dimensional historicimage data 140. Therefore, when the image retriever identifiescorresponding historic image 140, the point cloud containing bothhistoric images 140 and engineering model data 145 is retrieved.

Process 300 also includes key-point matching 320 to compare presentimages 115 to matching historic images 140. Key-point matching 320incorporates the extraction of specific features associated with imagesand uses the features to compare multiple images. In the exemplaryembodiment, features (not shown) of present images 115 are compared tofeatures (not shown) of historic images 140 that have been retrieved,resulting in pairs of matching features referring to the same physicallocation on physical asset 105. The features of present images 115 andhistoric images 140 may be the same features used in receiving 315 apoint cloud using the image retriever, or different features. In theexemplary embodiment, key-point matching 320 includes three steps.

First, key-point matching 320 compares images in terms of localappearance and local geometric patterns. Local appearance comparisonrepresents a comparison of a variety of image features. Local geometricpattern comparison represents a comparison of images as broken down tocomponent geometric features. This comparison further includesannotating historic images 140 with shape boundary information andlocating the same boundary in present images 115. This comparison alsoincludes the transformation of each feature location in historic images140 and present images 115. Transformation of the feature location canbe accomplished, for example, by producing a scale invariant featuretransformation (SIFT) for a feature on each image. The estimation methodalso includes comparing transformed features. In the exemplaryembodiment, comparing transformed features includes the application ofvalues to compensate for appearance and location of each feature,thereby adding location information to the transformed features.

Second, key-point matching 320 incorporates the use of an estimationmethod to find the best set of matched points between the images. Oncetransformed features have been compared, matches between features areobtained. The estimation method further includes applying two stepsusing random sample consensus (RANSAC) to eliminate outliers in thematches through multiple iterations. In the first step applying RANSAC,matches are constrained so that matching points must satisfy theepipolar constraint. To clarify, the epipolar constraint requires thatfor each point observed in one image the corresponding point must beobserved in the other image on a known epipolar line. Therefore, anepipolar line is determined for both present image 115 and historicimage 140 and used to create the epipolar constraint which is then usedto constrain matches. In at least some embodiments, the epipolarconstraint is determined by creating a fundamental matrix representativeof the epipolar line. Given the large number of matches, the RANSACapproach involves using a subset of initial matches that are randomlychosen from the complete set of matches. The merit of this subset ofinitial matches is determined by an error value computed as the level ofdisagreement with the epipolar constraint. Depending upon the number ofmatches in the complete set, the process is iterated until an adequatenumber of subsets are evaluated with respect to the epipolar constraint.The epipolar geometry associated with the subset that has the lowesterror rate is then used to eliminate matches that deviate from thisgeometry past a defined threshold.

In the second step applying RANSAC, matches are constrained to satisfy aprojection matrix. The projection matrix is computed using the retrievedpoint cloud. The projection matrix represents a mathematical matrix thatrelates the two-dimensional to three-dimensional point cloud that hasbeen transferred to present image 115. As in the first step, a subset ofinitial matches are randomly chosen from the set of matches (nowconstrained by the application of epipolar constraint). The projectionmatrix computed from this subset is assigned a merit value that dependsupon the agreement of the matches in this subset with the computedprojection matrix. The process is iterated depending upon the size ofthe total set of matches and the hypothesized projection matrix with thelowest error rate at the end is used.

Third, key-point matching 320 allows for associating engineering modeldata 145 retrieved together with historic image data 140 correspondingto present image data 115 by use of the retrieved point cloudcorresponding to historic image data 140. In this step, the associationor registration between two-dimensional historic image data 140 andthree-dimensional engineering model data 145 is transferred to presentimage data 115. Therefore, each point on present image data 115, ofwhich the match has been found, is now associated with three-dimensionalpoints on engineering model data 145.

Process 300 further includes camera pose estimation 325. Camera poseestimation 325 is representative of determining the specific positionand orientation of visual inspection device 110 with respect to physicalasset 105. Camera pose estimation 325 includes determining projectionsof potential camera locations based upon engineering model data 145associated with historic image data 140 and present images 115. In otherwords, engineering model data 145 is now associated or registered withpresent images 115 based upon the application of the point cloud(described above) and camera poses can be determined for present images115 by creating and applying a projection matrix to the mapping oftwo-dimensional present image 115 to three-dimensional engineering modeldata 145. In the exemplary embodiment, determining projections includessolving a system of linear equations representative of a projectionmatrix of potential camera poses.

Process 300 additionally includes rendering model to estimated camerapose 330. Rendering model to estimated camera pose 330 represents usingcamera pose estimation 325 on associated engineering model data 145 tocreate a rendered model representative of the engineering model from thesame camera position as visual inspection device 110.

Furthermore, process 300 includes physical measurement 335. Physicalmeasurement 335 represents comparison of physical features of therendered engineering model to present image 115. In the exemplaryembodiment, physical measurement 335 includes comparing image data froma projection of engineering model data 145 for a turbine blade topresent image 115 for a matching real turbine blade with a variance suchas a crack and measuring the crack.

FIG. 4 is a simplified flow chart of overall method 400 for enhancedvisual inspection of physical asset 105 using computer-implementedsystem 100 (both shown in FIG. 1) to facilitate enhanced visualinspection process 300 shown in FIG. 3. Computer device 120 (shown inFIG. 1) receives 415 at least one present image of physical asset 105.Receiving 415 at least one present image of physical asset 105represents receiving present visual inspection device data 305 (shown inFIG. 3) and, more specifically, receiving present images 115 from apresent image source. In the exemplary embodiment, the present imagesource is visual inspection device 110 (shown in FIG. 1). In alternativeembodiments, the present image source may be memory device 130, storagedevice 135, or external storage 160.

Computer device 120 also identifies 420 at least one matching historicimage corresponding to at least one present image. Identifying 420 atleast one matching historic image corresponding to at least one presentimage represents using image retriever 315 (shown in FIG. 3). Morespecifically, identifying 420 at least one matching historic imagecorresponding to at least one present image represents selectinghistoric image data 140 (shown in FIG. 1) that corresponds to presentimages 115.

Computer device 120 further identifies 425 at least one matchingengineering model corresponding to at least one present image.Identifying 425 at least one matching engineering model represents usingimage retriever 315 to receive a point cloud containing engineeringmodel data 145 (shown in FIG. 1) associated with historic image data140. As discussed above, the image retriever uses features determinedfor present image 115 and compares them to historic image data 140 todetermine historic image data 140 which most corresponds to presentimage 115. The computer-implemented systems and methods as describedherein provide an automated approach for inspecting physical assets witha visual inspection device. The embodiments described herein facilitateautomated comparison of image data from a visual inspection device withengineering model data without requiring manual intervention or externalinput. Due to the automatic retrieval of historic image, the automaticmatching between present and historic image features that achievestransferring of point cloud to the present image, the laborious processof manually picking feature matches between the model and the presentimage is avoided. Also, the methods and systems described hereinfacilitate rapid inspection and analysis of physical assets. Further,the methods and systems described herein will reduce the cost ofinspection through reduced time and human resources used for inspection.Additionally, these methods and systems will enhance the uptime ofphysical assets by reducing a need to disassemble an asset forinspection. Furthermore, the methods and systems described herein willreduce the operational, logistical, and financial costs associated withthe inspection of physical assets through efficient analysis and modelcomparisons.

An exemplary technical effect of the methods and computer-implementedsystems described herein includes at least one of (a) increases rate ofanalysis of visual data from a visual inspection device on physicalassets; (b) improved monitoring of physical assets through precisevisual data comparisons; and (c) greater uptime of physical assetsthrough a diminished need to disassemble the assets.

Exemplary embodiments for enhanced visual inspection of a physical assetare described above in detail. The computer-implemented systems andmethods of operating such systems are not limited to the specificembodiments described herein, but rather, components of systems and/orsteps of the methods may be utilized independently and separately fromother components and/or steps described herein. For example, the methodsmay also be used in combination with other enterprise systems andmethods, and are not limited to practice with only the visual inspectionsystems and methods as described herein. Rather, the exemplaryembodiment can be implemented and utilized in connection with many otherenterprise applications.

Although specific features of various embodiments of the invention maybe shown in some drawings and not in others, this is for convenienceonly. In accordance with the principles of the invention, any feature ofa drawing may be referenced and/or claimed in combination with anyfeature of any other drawing.

This written description uses examples to disclose the invention,including the best mode, and also to enable any person skilled in theart to practice the invention, including making and using any devices orsystems and performing any incorporated methods. The patentable scope ofthe invention is defined by the claims, and may include other examplesthat occur to those skilled in the art. Such other examples are intendedto be within the scope of the claims if they have structural elementsthat do not differ from the literal language of the claims, or if theyinclude equivalent structural elements with insubstantial differencesfrom the literal language of the claims.

What is claimed is:
 1. A computer-implemented system for enhancedautomated visual inspection of a physical asset comprising: a visualinspection device capable of generating images of the physical asset;and a computing device including a processor, a memory device coupled tosaid processor, and a storage device coupled to said memory device andto said processor, wherein said storage device includes at least onehistoric image of the physical asset and at least one engineering modelsubstantially representing the physical asset, said computing deviceconfigured to: receive, from a present image source, at least onepresent image of the physical asset captured by said visual inspectiondevice; identify at least one matching historic image corresponding tosaid at least one present image; and identify at least one matchingengineering model corresponding to said at least one present image.
 2. Acomputer-implemented system in accordance with claim 1, wherein saidpresent image source is at least one of: said visual inspection device;said storage device; said memory device; and an external storage device.3. A computer-implemented system in accordance with claim 1, whereinsaid computer-implemented system configured to identify said at leastone matching historic image corresponding to said at least one presentimage is further configured to: determine, from said at least onehistoric image, a set of historic salient features; determine, from saidat least one present image, a set of present salient features; anddetermine, using at least one image retrieval method, said at least onehistoric image corresponding to said at least one present image.
 4. Acomputer-implemented system in accordance with claim 3, furtherconfigured to use keypoint matching to identify the best sets ofmatching points between said at least one historic image and said atleast one present image.
 5. A computer-implemented system in accordancewith claim 1, wherein said computer-implemented system configured toidentify the at least one matching engineering model corresponding tosaid at least one present image is further configured to determine aphysical posture of said visual inspection device with respect to thephysical asset.
 6. A computer-implemented system in accordance withclaim 5, further configured to: determine two-dimensional pixellocations in said at least one matching historic image; determine, fromsaid two-dimensional pixel locations, three-dimensional point locationsin said matching engineering model; generate, from saidthree-dimensional point locations, a mathematical model of projectionsof said matching engineering model, said mathematical modelrepresentative of a function of characteristics of said visualinspection device, rotation of said visual inspection device, andtranslation of said visual inspection device; and determine, from saidmathematical model of projections of said matching engineering model andsaid at least one present image, a position of said visual inspectiondevice with respect to the physical asset.
 7. A computer-implementedsystem in accordance with claim 6, wherein said computer-implementedsystem configured to identify at least one matching engineering modelcorresponding to said at least one present image is further configuredto: create, using said position of said visual inspection device, aprojection of said matching engineering model; overlay said projectionof said at least one present image on said matching engineering model;and measure differences between said projection and said at least onepresent image.
 8. A computer-based method for enhanced automated visualinspection of a physical asset, wherein said method is performed by acomputing device, said computing device including a processor, a memorydevice coupled to said processor, and a storage device coupled to saidmemory device and to said processor, wherein said storage deviceincludes at least one historic image of the physical asset and at leastone engineering model substantially representing the physical asset,said method comprising: receiving, from a present image source, at leastone present image of the physical asset captured by said visualinspection device; identifying at least one matching historic imagecorresponding to said at least one present image; and identifying atleast one matching engineering model corresponding to said at least onepresent image.
 9. A computer-based method in accordance with claim 8,wherein said present image source is at least one of: said visualinspection device; said storage device; said memory device; and anexternal storage device.
 10. A computer-based method in accordance withclaim 8, wherein identifying said at least one matching historic imagecorresponding to said at least one present image comprises: determining,from said at least one historic image, a set of historic salientfeatures; determining, from said at least one present image, a set ofpresent salient features; and determining, using at least one imageretrieval method, said at least one historic image corresponding to saidat least one present image.
 11. A computer-based method in accordancewith claim 10, further comprising using keypoint matching to identifythe best sets of matching points between said at least one historicimage and said at least one present image.
 12. A computer-based methodin accordance with claim 8, wherein identifying the at least onematching engineering model corresponding to said at least one presentimage further comprises determining a physical posture of said visualinspection device with respect to the physical asset.
 13. Acomputer-based method in accordance with claim 12, further comprising:determining two-dimensional pixel locations in said at least onematching historic image; determining, from said two-dimensional pixellocations, three-dimensional point locations in said matchingengineering model; generating, from said three-dimensional pointlocations, a mathematical model of projections of said matchingengineering model, said mathematical model representative of a functionof characteristics of said visual inspection device, rotation of saidvisual inspection device, and translation of said visual inspectiondevice; and determining, from said mathematical model of projections ofsaid matching engineering model and said at least one present image, aposition of said visual inspection device with respect to the physicalasset.
 14. A computer-based method in accordance with claim 13, whereinidentifying at least one matching engineering model corresponding tosaid at least one present image further comprises: creating, using saidposition of said visual inspection device, a projection of saidengineering model; overlaying said at least one present image on saidengineering model; and measuring differences between said projection andsaid at least one present image.
 15. A computer for enhanced automatedvisual inspection of a physical asset, said computer includes aprocessor, a memory device coupled to said processor, and a storagedevice coupled to said memory device and to said processor, wherein saidstorage device includes at least one historic image of the physicalasset and at least one engineering model substantially representing thephysical asset, said computer configured to: receive, from a presentimage source, at least one present image of the physical asset capturedby said visual inspection device; identify at least one matchinghistoric image corresponding to said at least one present image; andidentify at least one matching engineering model corresponding to saidat least one present image.
 16. The computer of claim 15, wherein saidpresent image source is at least one of: said visual inspection device;said storage device; said memory device; and an external storage device.17. The computer of claim 16, further configured to use keypointmatching to identify the best sets of matching points between said atleast one historic image and said at least one present image.
 18. Thecomputer of claim 15, wherein said computer configured to identify atleast one matching engineering model corresponding to said at least onepresent image is further configured to determine a physical posture ofsaid visual inspection device with respect to the physical asset. 19.The computer of claim 18, further configured to: determinetwo-dimensional pixel locations in said at least one matching historicimage; determine, from said two-dimensional pixel locations,three-dimensional point locations in said matching engineering model;generate, from said three-dimensional point locations, a mathematicalmodel of projections of said matching engineering model, saidmathematical model representative of a function of characteristics ofsaid visual inspection device, rotation of said visual inspectiondevice, and translation of said visual inspection device; and determine,from said mathematical model of projections of said matching engineeringmodel and said at least one present image, a position of said visualinspection device with respect to the physical asset.
 20. The computerof claim 19, wherein said computer configured to identify at least onematching engineering model corresponding to said at least one presentimage is further configured to: create, using said position of saidvisual inspection device, a projection of said engineering model;overlay said at least one present image on said engineering model; andmeasure differences between said projection and said at least onepresent image.